Subspace techniques for multidimensional model order selection in colored noise

R-dimensional (R-D) harmonic retrieval (HR) in colored noise, where RZ2, is required in numerous applications including radar, sonar, mobile communications, multiple-input multiple-output channel estimation and nuclear magnetic resonance spectroscopy. Tensor-based subspace approaches to R-D HR such as R-D unitary ESPRIT and R-D MUSIC provide super-resolution performance. However, they require the prior knowledge of the number of signals. The matrix based (1-D) ESTimation ERror (ESTER) is subspace based detection method that is robust against colored noise. To estimate the number of signals from R-D measurements corrupted by colored noise, we propose two R-D extensions of the 1-D ESTER by means of the higher-order singular value decomposition. The first R-D ESTER combines R shift invariance equations each applied in one dimension. It inherits and enhances the robustness of the 1-D ESTER against colored noise, and outperforms the state-of-the-art R-D order selection rules particularly in strongly correlated colored noise environment. The second R-D scheme is developed based on the tensor shift invariance equation. It performs best over a wide range of low-to-moderate noise correlation levels, but poorly for high noise correlation levels showing a weakened robustness to colored noise. Compared with the existing R-D ESTER scheme, both proposals are able to identify much more signals when the spatial dimension lengths are distinct. & 2013 Elsevier B.V. All rights reserved.